[ci] Add GPG-signed DEB packages#5180
Open
PlatCore wants to merge 41 commits into
Open
Conversation
1 task
maru-ava
reviewed
Apr 6, 2026
maru-ava
reviewed
Apr 6, 2026
maru-ava
reviewed
Apr 6, 2026
maru-ava
reviewed
Apr 6, 2026
maru-ava
reviewed
Apr 6, 2026
c6cdc18 to
84aa1f4
Compare
maru-ava
reviewed
Apr 8, 2026
Contributor
maru-ava
left a comment
There was a problem hiding this comment.
I'm intentionally keeping this review narrow until 5179 is ready. That said, there is at least one DEB-specific correctness issue here that seems worth calling out now: the release-signing path appears to rely on passphrase handling that PR CI does not exercise, and the current gpg-preset-passphrase invocation does not look correct.
f2c029c to
402dd0c
Compare
yacovm
reviewed
Apr 12, 2026
402dd0c to
d336cf3
Compare
5f4fb6a to
e9e97d1
Compare
be87574 to
e03431d
Compare
Contributor
Author
|
The setup-packaging Taskfile task was removed and replaced by workflow-setup-packaging.sh. The RPM workflow was updated but the DEB workflow still referenced the old task name.
RPM refactor. After rebasing this branch onto the post-excision tip of #5179, the hooks need to return as commits owned by #5180 so they survive the eventual merge into master. build-package.sh: source lib-build-${fmt_lower}.sh on FORMAT=DEB, add the DEB arm to the GPG_KEY_FILE case, configure gpg-agent via setup_deb_gpg_agent before setup_gpg, cache the passphrase via cache_deb_gpg_passphrase for non-interactive dpkg-sig signing, and invoke sign_deb_package post-nfpm (nfpm's openpgp signatures are incompatible with dpkg-sig --verify). lib-validate-common.sh: re-add the DEB arm of detect_host_arch's format dispatch (x86_64 -> amd64, aarch64/arm64 -> arm64). The corresponding helper functions are defined in lib-build-deb.sh. Verified locally: task -t .github/packaging/Taskfile.yml test-build-debs passes end-to-end. Both .debs built, dpkg-sig signatures verify (GOODSIG), and install + smoke test pass in fresh ubuntu:22.04 (jammy) and ubuntu:24.04 (noble) containers.
Mirrors the RPM template refactor. build-package.sh exports only
${BINARY_PATH}; the DEB templates' references to
${AVALANCHEGO_BINARY} / ${SUBNET_EVM_BINARY} resolved to empty via
envsubst, causing nfpm to fail with 'glob failed: no matching files'
on the amd64 CI run.
actionlint flagged 'custom-arm64-jammy' as an unknown label; master migrated arm64 to the GitHub-hosted ubuntu-22.04-arm runner. The DEB workflow was authored before that sweep.
nfpm now signs DEBs inline via deb.signature.key_file, matching the RPM side. Validation extracts the _gpgorigin ar member and runs gpg --verify against debian-binary+control+data, so the same flow works on jammy and noble (dpkg-sig was unavailable in noble). Builder image, build helper, Taskfile comment, and design doc no longer reference dpkg-sig.
- workflow-setup-packaging.sh now fails fast when no signing key is
provided for a tag push or workflow_dispatch (previously fell back
to ephemeral-key generation, so unsigned-by-real-key packages could
reach S3).
- build-{deb,rpm}-release.yml replace .github/packaging atomically
(rm -rf + mv) instead of `cp -r` into an existing directory, which
was leaving the checked-out tag's stale Taskfile in place and
copying the overlay to .github/packaging/packaging/.
- resolve_subnet_evm_vm_id falls back to grepping
graft/subnet-evm/scripts/constants.sh when the new
default-vm-data.sh is absent, restoring workflow_dispatch builds of
tags predating the data file.
- Taskfile.yml forwards NFPM_{RPM,DEB}_PASSPHRASE via a task-level
`env:` block and value-less `-e NFPM_*_PASSPHRASE` so passphrases
containing whitespace or shell metacharacters reach the container
intact (no shell evaluation of the secret).
Taskfile.yml: the four build-{avalanchego,subnet-evm}-{rpm,deb} tasks
collapse into one internal `build-package` task that the four wrappers
delegate to with `task: build-package` + vars. The wrappers only carry
the format/output/builder-image differences; the docker-run block lives
in one place.
build-package.sh: switch the format env var from PKG_FORMAT (RPM|DEB) to
NFPM_PACKAGER (rpm|deb) to match nfpm's CLI naming, and collapse
RPM_GPG_KEY_FILE / DEB_GPG_KEY_FILE into a single GPG_KEY_FILE since the
caller already picks the right one.
The value-less `-e NFPM_*_PASSPHRASE` passphrase forwarding survives the
refactor (verified via task --dry with a metachar-laden passphrase).
Moves VERSION (derived from TAG via trimPrefix "v") and TAG into the build-package task's env: block and switches their docker flags to the value-less `-e VAR` form, matching the pattern already used for the NFPM_*_PASSPHRASE secrets. Keeps the docker invocation free of any template-substituted values that could carry shell metacharacters.
Aligns the RPM-side var names with the DEB-side pattern (format prefix between PACKAGING_ and the rest): PACKAGING_HOST_ARCH -> PACKAGING_RPM_HOST_ARCH PACKAGING_HOST_DEB_ARCH -> PACKAGING_DEB_HOST_ARCH PACKAGING_DOCKER_IMAGE -> PACKAGING_RPM_DOCKER_IMAGE PACKAGING_OUTPUT_DIR -> PACKAGING_RPM_OUTPUT_DIR All references inside the Taskfile updated; no live consumers outside it.
Replaces docs/design/deb-packaging.md and docs/design/rpm-packaging.md with a single shared design doc per reviewer comment on PR #5180. The new doc has format-specific subsections for RPM and DEB and shared subsections for build tooling, GPG signing, version smoke test, CI workflow, and architecture mapping. Also brings the prose in line with the current code: - VM-ID source notes default-vm-data.sh with a fallback to constants.sh for older tags, matching the resolver in lib-build-common.sh. - CI workflow examples use the actual --taskfile invocation form used in build-{rpm,deb}-release.yml, with a note about the packaging: namespace available locally via the root Taskfile include. - GPG-signing section documents the fail-fast release-key gate that workflow-setup-packaging.sh enforces for tag pushes and workflow_dispatch.
The `.github/actions/setup-packaging/**` composite-action path was removed from this branch, so the pull_request.paths filter referencing it never matches. Drop it from both build-rpm-release.yml and build-deb-release.yml.
Both release workflows pipe the same `gpg-key-file` step output into
the build env, and the per-format wrappers immediately funnel that into
build-package's GPG_KEY_FILE var. The {RPM,DEB}_ prefix carried no
information.
- Taskfile: four wrappers now read .GPG_KEY_FILE directly instead of
the format-prefixed equivalents.
- build-deb-release.yml: env block sets GPG_KEY_FILE (was
DEB_GPG_KEY_FILE) to match the already-normalized RPM workflow.
- build-deb-release.yml: GPG_KEY_PASSPHRASE replaces NFPM_DEB_PASSPHRASE
(the latter was no longer read by the Taskfile; releases were silently
unsigned) and is gated on non-pull_request like the RPM side.
NFPM_RPM_PASSPHRASE / NFPM_DEB_PASSPHRASE remain *inside*
build-package.sh as the nfpm-mandated env var names; the script
re-exports GPG_KEY_PASSPHRASE under the right NFPM_<FORMAT>_PASSPHRASE
just before invoking nfpm.
The RPM builder-image task was the only build-* task that omitted the format prefix. With a parallel build-deb-builder-docker-image already in place, the RPM task is now build-rpm-builder-docker-image so every build-* task carries the same format suffix (or is the format-agnostic internal build-package). - Task definition at line 59. - deps: refs in build-avalanchego-rpm and build-subnet-evm-rpm. The underlying scripts/build-builder-image.sh stays unrenamed — it is already format-agnostic (drives off the DOCKERFILE arg).
build-package.sh sets set -euo pipefail. When the Taskfile invokes
build-package without a real signing key, the docker run omits the
`-e GPG_KEY_FILE` flag entirely (Task conditional), so the container env
is unset. The script then references ${GPG_KEY_FILE} in the ephemeral-
key branch and aborts on `unbound variable` before signing setup runs.
Default GPG_KEY_FILE to empty alongside the existing NFPM_*_PASSPHRASE
defaults; the ephemeral-key conditional handles the empty case correctly.
f946cc6 renamed PACKAGING_HOST_ARCH to PACKAGING_RPM_HOST_ARCH but missed the default chain in the validate-rpms task. With no explicit PACKAGE_ARCH (the normal CI / test-build-rpms path), PACKAGE_ARCH resolves to empty and validate-rpm.sh aborts at its `: "${PACKAGE_ARCH:?...}"` assertion -- after the packages have been built.
build-package.sh now exports the public key as
${pkg_format_upper}-GPG-KEY-avalanchego, so the RPM build produces
build/rpm/RPM-GPG-KEY-avalanchego. The two consumers still referenced
the unprefixed path:
- validate-rpm.sh checked /rpms/GPG-KEY-avalanchego and silently skipped
signature verification on signed RPMs.
- build-rpm-release.yml uploaded build/rpm/GPG-KEY-avalanchego, so the
release artifact never included the key.
Mirror the DEB side (which already uses DEB-GPG-KEY-avalanchego
end-to-end).
build-deb runs on pull_request and executes PR-controlled packaging scripts. Granting id-token: write at the job level let PR scripts request a GitHub OIDC JWT regardless of which steps actually ran. Split into two jobs: - build-deb: contents: read only. Builds, validates, and uploads artifacts on non-PR runs. Exposes the resolved tag as a job output. - upload-debs-s3: needs: build-deb; runs only on tag push or workflow_dispatch; has id-token: write. Downloads artifacts and pushes them to S3. The PR-executed job can no longer mint OIDC tokens; release uploads keep working via the new dedicated job.
resolve_subnet_evm_vm_id sources constants.sh inside SUBNET_EVM_VM_ID=$(...)
so it captures whatever stdout the sourced file produces along with the
trailing echo "${DEFAULT_VM_ID}".
Older subnet-evm constants.sh revisions (e.g., v1.14.1 line 62) do
`echo "Using branch: ${CURRENT_BRANCH}"` at source time. Under
workflow_dispatch tag-overlay builds this contaminates SUBNET_EVM_VM_ID
with a "Using branch: ..." line, which then corrupts the rendered nfpm
yaml plugin path.
Redirect the source's stdout to /dev/null; the explicit
`echo "${DEFAULT_VM_ID}"` at the end of the subshell is the only thing
captured.
PR #5179 (Thread #7, pushed as 7245fa9) shrank smoke-test.sh from 4 args to 3: the caller now passes the composed subnet-evm plugin binary path instead of the plugin dir + VM ID. validate-rpm.sh was updated in the same commit; validate-deb.sh was missed and kept the old 4-arg invocation. After rebasing onto PR #5179's tip the contract drift surfaced in build-deb-packages CI: positional-arg slip made $2 the plugins *directory*, and smoke-test.sh tried to exec it ("Is a directory", exit 126). Mirror the validate-rpm.sh fix.
Four regressions from PR #5179's review outcomes were carried into the DEB-side code on PR #5180: R1 — build-package.sh: remove the "Required/Optional env vars" header docstring reintroduced by the NFPM_PACKAGER refactor. The ${VAR:?msg} asserts already carry inline role descriptions (PR #5179 Threads #9/#10). R2 — validate-deb.sh: same header-docstring removal. Add PACKAGE_ARCH as a required assert with role description, mirroring validate-rpm.sh (PR #5179 Threads #10/#11). R3 — validate-deb.sh: convert the "Skipping GPG verification (unsigned build)" else branch to a fatal error. The pipeline always produces a signed key (PR #5179 Thread #8). R4 — Delete lib-validate-common.sh (reintroduced during the PR #5180 rebase with detect_host_arch). The Taskfile already computes host arch via PACKAGING_DEB_HOST_ARCH; validate-debs now passes PACKAGE_ARCH through its env block, mirroring validate-rpms (PR #5179 Thread #6).
Concrete fixes:
- subnet-evm-deb.yml: fix stale comment referencing non-existent
default-vm-data.sh; the actual source is constants.sh (matches
the RPM config and resolve_subnet_evm_vm_id in lib-build-common.sh).
- avalanchego-deb.yml + subnet-evm-deb.yml: bump libc6 dependency
from >= 2.34 to >= 2.35 to match the DEB builder's Ubuntu 22.04
glibc version (2.34 is the RPM/Rocky 9 floor).
- build-package.sh: remove redundant static exports of
NFPM_RPM_PASSPHRASE and NFPM_DEB_PASSPHRASE; the dynamic export
on line 49 already sets the correct format-specific var.
- Taskfile.yml:
- Remove undefined .RPM_ARCH / .DEB_ARCH from PACKAGE_ARCH
default chains (never set by any caller; dead chain links).
- Move PACKAGING_DEB_HOST_ARCH adjacent to PACKAGING_RPM_HOST_ARCH
for easy comparison.
- Replace `sh: echo "${PACKAGING_TAG:-v0.0.0}"` with Go template
`{{env "PACKAGING_TAG" | default "v0.0.0"}}` (no shell spawn).
- Sort tasks alphabetically so RPM/DEB equivalents are
side-by-side, per reviewer request for sorted ordering.
- Remove unused PASSPHRASE_ENV var from build-package task.
Verified: test-build-debs and test-build-rpms both pass end-to-end.
Move the DEB + RPM design content from docs/design/linux-packaging.md into the canonical .github/packaging/README.md per reviewer comment and the convention established in #5373. Fixed stale details while moving: constants.sh VM ID source and libc6 (>= 2.35) DEB dependency floor.
Addresses the lightweight code-side feedback from the 2026-05-27 review batch on #5180. Functional behavior unchanged; verified end-to-end via `task -t .github/packaging/Taskfile.yml test-build-rpms` and the DEB counterpart. - build-package.sh: restore the `${PACKAGE:?…}` assert (was missing entirely) and enrich all asserts with inline role descriptions, matching the inline-assert convention from #5179 Thread #10. Drops the prefix from the exported public key (`GPG_PUBLIC_KEY` is now unprefixed — see below). - lib-build-common.sh: drop the redundant `>/dev/null` and the misleading comment in `resolve_subnet_evm_vm_id`. The grouped block already routes its stdout to stderr via `>&2`, so the inner redirect was a no-op and the comment described a problem the existing redirect already handled. - GPG-KEY filename: drop the format prefix end-to-end. RPM and DEB builds already separate outputs by directory (`build/rpm/` vs `build/deb/`, `/rpms/` vs `/debs/`, separate workflow artifact paths and S3 prefixes), so the per-format prefix was redundant. Updated build-package.sh, validate-rpm.sh, validate-deb.sh, build-rpm-release.yml (artifact upload include), build-deb-release.yml (artifact upload include + S3 `aws s3 cp` source), and the README. - Dockerfile.{rpm,deb}: drop `=INVALID` from `GO_VERSION` and `GO_CHECKSUM` ARGs. Post-`FROM` ARGs don't benefit from the sentinel default — an unset value would behave identically (the `sha256sum -c` step catches the bad download in either case). - Taskfile.yml: rename `build-{rpm,deb}-builder-docker-image` → `build-builder-docker-image-{rpm,deb}` so the task names sort consistently with the other format-suffixed tasks. Reordered the two task definitions to match the alphabetical order. Added an explicit `requires:` block to the internal `build-package` task surfacing its API contract (PACKAGE, NFPM_PACKAGER, TAG, PACKAGE_ARCH, OUTPUT_DIR, DOCKER_IMAGE). - validate-deb.sh: drop the stale dpkg-sig advocacy sentence from the `_gpgorigin` verification comment. dpkg-sig isn't used; the comment now describes only what we actually do. - build-deb-release.yml: add the top-level `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true` env block that the RPM workflow and the deleted DEB workflows already set. Without it, Node-20 JS actions (`actions/checkout`, the setup-go composite, artifact actions, AWS credentials action) can fail during action setup in Node 24-gated environments before packaging starts. Addresses - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306495025 - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306509777 - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306532461 - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306542567 - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3167909431 - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3310777624 - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3310801323
Per maru-ava feedback on thread #5180 (build-deb-release.yml:1 + workflow-setup-packaging.sh:44): the post-overlay steps for RPM and DEB are nearly identical, and the release-key gate was duplicated between the workflow YAML and the setup script. Factor both into a shared composite action so common behavior can't diverge. - Add `.github/packaging/actions/build-package/action.yml` — composite action that handles setup-go, workflow-setup-packaging, build-and-validate (task test-build-${format}s), artifact upload (gated on release builds), and cleanup. Format/arch differences come in as inputs. Lives under .github/packaging/ so the workflow-branch overlay step (used for manual rebuilds of older tags) makes it available alongside the rest of the packaging tree. Approach validated by maru-ava's PoC in #5429. - Add `.github/workflows/build-linux-packages.yml` — single workflow with a format × arch matrix delegating bulk work to the composite action. The DEB-only S3 upload remains a separate release-only job (id-token: write stays out of the build job). RPM S3 publishing can land as a sibling job later when the RPM-side S3 path is defined. - Delete `.github/workflows/build-{rpm,deb}-release.yml` — both are subsumed by build-linux-packages.yml. - workflow-setup-packaging.sh: drop the event-type sniffing block (TAG_INPUT / GITHUB_REF tag-check). The RELEASE-flag check below remains the single source of truth for release-mode gating, and the new composite action sets RELEASE for both formats from one place so the two checks can't diverge. Verified locally: - `nix develop --command task test-build-rpms` → green - `nix develop --command task test-build-debs` → green - workflow-setup-packaging.sh smoke tests pass on release-with-key (success), release-without-key (fatal as expected), and PR (success, no key, no error) - actionlint clean on build-linux-packages.yml Note for reviewer: branch protection rules referencing the old status check names (build-rpm-packages / build-deb-packages) will need to be updated to the new build-linux-packages name before merge. Addresses - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306572758 - https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306538214
Three locations in `.github/packaging/README.md` still named the per-format workflows that (b254f0df89) deleted. Replace them with references to the unified `build-linux-packages.yml` + the new `build-package` composite action. - "Main entrypoints" bullet block - "CI behavior" intro sentence - "Repository entrypoints" bullet block
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this should be merged
DEB packages are currently unsigned and built with an ad-hoc
dpkg-debflow. This adds GPG-signed DEB packages foravalanchegoandsubnet-evm, modelled on the existing RPM pipeline — containerized build, nfpm-native signing, end-to-end validation on both supported Ubuntu releases.Depends on #5179.
How this works
Mirrors the RPM pipeline pattern with a DEB-specific container:
Dockerfile.deb— Ubuntu 22.04 builder image (gcc, gnupg, Go, nfpm).build-package.sh+lib-build-common.sh— unified RPM/DEB build helper. Builds the binary, sets up GPG (ephemeral key for PR/local;secrets.RPM_GPG_PRIVATE_KEYfor releases), and packages with nfpm.deb.signature.key_file: "${NFPM_SIGNING_KEY}"so nfpm signs in-process (no post-build signing step). Install paths:/usr/local/bin/avalanchegoand/usr/local/lib/avalanchego/plugins/<VM_ID>.validate-deb.sh— in bothubuntu:22.04andubuntu:24.04: extracts_gpgoriginfrom the ar archive, runsgpg --verifyagainstdebian-binary + control.tar.* + data.tar.*, then installs the packages and runs the shared smoke test.Taskfile.yml— Docker-run-based DEB tasks matching the RPM task pattern.task packaging:test-build-debsworks locally on macOS.build-deb-release.yml— invokestask packaging:test-build-debs. Uploads artifacts on non-PR runs and pushes to S3 on tag push /workflow_dispatch.workflow-setup-packaging.shfails fast if the signing key secret is absent on a release event.build-ubuntu-amd64-release.yml,build-ubuntu-arm64-release.yml,build-deb-pkg.sh,debian/template/control.Key design decisions:
_gpgoriginar member) verified withgpg --verify. Works identically on jammy and noble — nodpkg-siganywhere.secrets.RPM_GPG_PRIVATE_KEY; the passphrase is forwarded into the container with a value-less-e VARflag so secrets with whitespace or shell metacharacters reachnfpmintact.How this was tested
task packaging:test-build-debslocally on macOS (Docker): both packages built and signed by nfpm,gpg --verifygood in jammy and noble containers, install + smoke test passed on both.task --dryof the build task confirming the passphrase is no longer interpolated into the docker command-line..github/packaging/**.Need to be documented in RELEASES.md?
No